NC STATE UNIVERSITY 


Updated Results for the North Carolina 
Read to Achieve Program 

Implications and Next Steps for the State 

Trip Stallings, Sara Weiss, & Steve Porter 

North Carolina State University 

North Carolina State Board of Education 

January 9, 2019 

This research project has been funded by the Institute for Education Sciences under the Low-Cost, Short-Duration Evaluation of 
Education Interventions program, grant number R305L160017. The opinions expressed are those of the authors and do not 
represent views of the Institute or the U.S. Department of Education. 



NC STATE UNIVERSITY 


Agenda 

• Summary 

• Results 

• Gaps between Policy and Implementation 

• Moving Forward: What the State Can Do Next 



NC STATE UNIVERSITY 




NC STATE UNIVERSITY 


Summary 

• Statewide impact is null one year out (what would have 
been students’ 4**^ grade year) and two years out - 1314 
& 1415 cohorts 

o All initially-impacted students (whether eventually retained or not) 
o Retained students only 

• Summer camp attendance did not impact 4'*^ grade 
outcomes (1415 cohort) 

• In practice, RtA appears to be 115 different pilots 
operating under a few common parameters 
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Results 


There does not appear to be any noticeable gain for students below 
the cut-point, whether we look at Cohort 1 or Cohort 2, 1 year out or 2 
years out: 




2014-15 cohort, one year out 2013-14 cohort, two years out 

There does not appear to be any effect by sub-group. 

On a positive note, math scores do not appear to have been impacted. 
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Results 

Did most students at least do better than they would have without 
the intervention? To answer that, we focus on the cut-point .... 


0000 


3rd grade reading EOG score {2013-14) 

2013-14 cohort, two years out 
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Results 

What about Reading Camps? 

• Four groups of students who failed initial 3'^'^ grade EoG: 

• Passed a subsequent reading EOG or alternate test and were 
promoted 

• Did not pass a subsequent reading EOG or alternate test, attended 
reading camps, and then were promoted 

• Did not pass a subsequent reading EOG or alternate test, attended 
reading camps but did not pass, and then were retained 

• Did not pass a subsequent reading EOG or alternate test, did not 
attend reading camp, and then were retained 
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Results 

Why no apparent effect? 

• Analyses are state-level only*, and only consider 
the impact of the overall policy, not the impact of 
specific LEA- and school-level interventions 

• Analyses do not estimate effects on lowest- 
performing students*. 

• But also _ 
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Gaps between Policy & Implementation 


Policy 

• No support for pre-3'^'^ 
grade interventions 
{partially addressed 
for later cohorts) 

• Broad definition of 
reading proficiency 
(e.g., exemptions) 

• Assumptions re: local 
capacity (differenti¬ 
ation, human capacity) 


Implementation 

• Local camp decisions 
(length, timing, 
staffing) 

• Variable capacity (both 
in terms of people and 
funds) 

• Variations in 4*^^ grade 
year placement 

• Variations in retained 
student experiences 



North Carolina Third Graders 
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Gaps between Policy & Implementation 

Example'. Broad definition of reading proficiency 
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■ Failed Initial Reading EOG 


First Reading EOG, 
Spring 2015 
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Gaps between Policy & Implementation 

Example: Broad definition of reading proficiency 

• About half of initially non-proficient students who were 
promoted before the next school year were promoted via a 

local assessment 

• These students performed better the following year relative 
to students who were not promoted; however... 

• They did not perform as well as initially-identified students 
who were promoted via all other measures 
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Gaps between Policy & Implementation 

Example: Variations in retained student experiences 

• Of schools that responded to a survey (about one-third) 
o 3rd grade/4th grade transition class: 65% 

o 4th grade accelerated reading class: 42% 
o Traditional 3rd grade repetition: 25% 
o 3rd grade accelerated reading class: 9% 

• Outcomes by placement varied 

• Placement in a 3'"^ grade accelerated reading class: 
statistically significant, positive outcome (relative to 
traditional 3'^'^ grade retention) 

• Caveat. Analyses were for a handful of students from a 
non-random sample of schools 
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Moving Forward: What the State Can Do Next 

• Collect more extensive implementation data 
(including individual student placement data and 
better summer camp data) 

• Focus on implementation fidelity 

• Identify and scale up local successes 

o Requires replication and expansion of original 
study to include 2015-16 cohort and beyond 

• Extend and expand pre-S'^'^ grade literacy 
development 



NC STATE UNIVERSITY 


Contact Information 

Trip Stallings 
Director of Policy Research 
Friday Institute for Educational Innovation, NCSU 

dtstalli@ncsu.edu 

Sara Weiss 
Senior Research Scholar 
Friday Institute for Educational Innovation, NCSU 

stpilzer@ncsu.edu 

Steve Porter 

Professor, Dept, of Leadership, Policy and Adult & Higher Education 

College of Education, NCSU 
srporter@ncsu.edu 






NC STATE UNIVERSITY 


Overview of Read to Achieve 

Goal 

• Ensure that all students read at or above grade level by the end of grade 3. 

Determination of Proficiency 

• Initial Measure: Passing score (score of III, IV, or V) on Grade 3 Reading EoG 

• Additional Measures: EoG re-takes; Grade 3 BoG; RtAtest; approved local 
assessments; portfolios 

• Good-Cause Exemptions: LEP, EC identification, previous retention, etc. 

interventions and Supports 

• Reading Camps: Optional remediation (at least 72 hours) with high-quality 
reading teachers for students without good-cause exemptions or successful 
re-tests 

• Following-Year Reading-Intensive Class Placements: 

• Grade 3 with accelerated reading component 

• Hybrid Grade 3/Grade 4 transition class with additional reading instruction 

• Grade 4 setting with pull-out reading instruction 
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Can the Policy Work? 

Grade 3 Retention and Student Achievement 

• Evidence of short-term positive effects in higher-quality studies 
some evidence of longer-term carry-over 

Student Motivation and Negative Consequences 

• Unclear; some evidence that older students work harder, but 
without positive academic results 

Summer School and Student Achievement 

• Evidence of positive gains, tempered by program quality 

Teacher Quality and Student Achievement 

• As defined by value-added metrics, teacher quality matters 

Additional Reading Instruction and Student Achievement 

• Unclear, but promising if sustained across grades 
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Results 


Research Questions 

1. What is the causal effect of the Read to Achieve program on 
subsequent student reading performance one year and two 
years later? 

2. What is the causal effect of the Read to Achieve program on 
promotion to grade four and grade five? 

3. How do short- and longer-term effects vary by student sub¬ 
groups (e.g., gender, race/ethnicity, and economic 
disadvantage, means of demonstrating proficiency)? 
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Retention 
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Results 

Important Context/Frame for our Study: 

• To determine causal, not correlational, outcomes, we need 
to focus on students who are nearly identical in ail ways, 
except that some are exposed to RtA and others are not 

• To do that, we pay special attention to students immediateiy 
on either side of the Grade 3 EoG score eiigibility line— the 

scores just below and Just above the “proficient” line 

• Students on either side of this line, as a group, shouid not be 
different from each other in any other way; most are affected 
by RtA by random chance (e.g., some guessed right on one 
question, others guessed wrong) 
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Results 


Our Target Groups 



3rd grade reading EOG score (2013-14) 


Note: The lower a student’s initial score, the more likely that she or he ended up 

being retained. 
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The Gap between Policy & Implementation 

Pathways to Promotion after Initial EOG Failure (Y2) 


Passed at Least 1 retest 
Passed Local Assessment 
Passed RtA Exam, Spring '15 
Passed Portfolio 
Passed EOG Retest 
Passed BOG 
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Passed RtA Exam after 2015 Camp 
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